from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.129	0.280	0.000	0.002	-1	1	0.663	17.337	0.047	0.663	0.123	0.123	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	1	1.000	0.313	0.002	1.000	0.075	0.075	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.824	0.029	0.000	0.003	-1	5	0.757	17.081	0.178	0.757	0.165	0.165	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.003	0.000	0.023	-1	5	1.000	0.314	0.004	1.000	0.074	0.074	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.107	0.091	0.000	0.002	1	100	0.882	17.063	0.054	0.882	0.123	0.123	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	100	1.000	0.317	0.006	1.000	0.060	0.060	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.871	0.159	0.000	0.003	-1	100	0.882	17.302	0.138	0.882	0.166	0.166	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	100	1.000	0.313	0.003	1.000	0.077	0.077	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.059	0.014	0.000	0.002	1	5	0.757	17.465	0.175	0.757	0.118	0.118	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.001	0.000	0.020	1	5	1.000	0.307	0.008	1.000	0.065	0.065	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.199	0.005	0.001	0.001	1	1	0.663	17.547	0.110	0.663	0.068	0.068	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	1	1.000	0.319	0.005	1.000	0.059	0.059	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.807	0.034	0.000	0.002	-1	1	0.896	3.776	0.014	0.896	0.479	0.479	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	0.000	0.006	-1	1	1.000	0.238	0.003	1.000	0.024	0.024	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.847	0.101	0.000	0.003	-1	5	0.922	3.692	0.024	0.922	0.771	0.771	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.002	0.000	0.007	-1	5	1.000	0.236	0.003	1.000	0.028	0.028	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.965	0.012	0.000	0.002	1	100	0.929	3.762	0.012	0.929	0.522	0.522	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	100	1.000	0.239	0.004	1.000	0.012	0.012	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.757	0.043	0.000	0.003	-1	100	0.929	3.756	0.010	0.929	0.734	0.734	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.003	0.000	0.006	-1	100	1.000	0.239	0.004	1.000	0.025	0.025	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.940	0.005	0.000	0.002	1	5	0.922	3.730	0.009	0.922	0.520	0.520	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	5	1.000	0.239	0.004	1.000	0.012	0.012	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.094	0.008	0.000	0.001	1	1	0.896	3.778	0.007	0.896	0.289	0.289	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.240	0.005	1.000	0.008	0.008	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.822	0.946	0.000	0.001	-1	1	0.929	105.283	0.000	0.929	0.008	0.008	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.002	0.000	0.003	-1	1	1.000	2.619	0.163	1.000	0.001	0.001	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.982	0.301	0.000	0.001	-1	5	0.946	105.170	0.000	0.946	0.009	0.009	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	1.000	2.616	0.180	1.000	0.001	0.001	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.425	0.347	0.000	0.005	1	100	0.951	107.860	0.000	0.951	0.050	0.050	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	1.000	2.609	0.146	1.000	0.001	0.001	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.143	0.243	0.000	0.003	-1	100	0.951	106.993	0.000	0.951	0.029	0.029	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	1.000	2.627	0.141	1.000	0.002	0.002	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.581	0.122	0.000	0.002	1	5	0.946	106.126	0.000	0.946	0.015	0.015	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.001	0.000	0.002	1	5	1.000	2.604	0.078	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.873	0.101	0.000	0.001	1	1	0.929	105.090	0.000	0.929	0.008	0.008	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	2.596	0.120	1.000	0.000	0.000	See	See
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.026	0.013	0.001	0.000	-1	1	0.891	0.045	0.013	0.891	0.582	0.607	See	See
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.001	0.000	0.002	-1	1	1.000	0.005	0.000	1.000	0.482	0.483	See	See
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.022	0.001	0.001	0.000	-1	5	0.911	0.037	0.001	0.911	0.600	0.600	See	See
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	1.000	0.005	0.000	1.000	0.376	0.376	See	See
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.035	0.006	0.000	0.000	1	100	0.894	0.058	0.001	0.894	0.598	0.598	See	See
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.005	0.000	1.000	0.104	0.104	See	See
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.036	0.003	0.000	0.000	-1	100	0.894	0.058	0.001	0.894	0.628	0.628	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	100	1.000	0.005	0.000	1.000	0.373	0.373	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.020	0.000	0.001	0.000	1	5	0.911	0.038	0.000	0.911	0.526	0.527	See	See
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.005	0.000	1.000	0.106	0.106	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.019	0.000	0.001	0.000	1	1	0.891	0.038	0.000	0.891	0.501	0.501	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.005	0.000	1.000	0.107	0.107	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.107	0.001	300	0.007	0.000	0.824	0.496	0.023	0.824	0.215	0.216	See	See
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.016	0.000	300	0.000	0.016	1.000	0.413	0.008	1.000	0.039	0.039	See	See